Add pm-gpu #835
Add pm-gpu #835
Conversation
xylar
added
the
dependencies and deployment
|
@mcarlson801 and @jewatkins, this is the starting point for adding pm-gpu support (with I have been able to run the |
|
A note to say that I tried to build Albany and Trilinos with |
|
@mcarlson801 and @jewatkins, I was able to build the trilinos and albany spack libraries (and the rest of the compass spack environment) using this branch, https://github.com/xylar/mache/tree/add-cuda-to-pm-gpu and @mcarlson801's E3SM-Project/spack#31. I was also able to build MALI from the I ran the |
|
The following restart tests failed with a validation error: The errors are all quantitatively similar to the following: |
|
The |
|
Here's the timing I'm seeing: |
Did you build Albany with the +slfad variant? I think this is the same error I ran into when I was building with DFad (although other tests would be failing with it too, hmmm). I'll take a look and see what's up. |
No, I missed that. Is that for Trilinos? Albany? both? |
|
Actually, scratch that, for MALI we would use |
|
Okay, thanks, I'll add that and try again. |
|
Actually, to specify sfad 12, I think it's probably This is the line where the sfadsize gets used: https://github.com/E3SM-Project/spack/blob/develop/var/spack/repos/builtin/packages/albany/package.py#L130 And this is the line where the sfadsize is obtained: https://github.com/E3SM-Project/spack/blob/develop/var/spack/repos/builtin/packages/albany/package.py#L46 @ikalash Do you know what we need to add to the variants to install with |
|
@mcarlson801 please keep me posted, then. |
|
I looked up how to set multi-valued variants and it looks like the way to do this would be to add |
|
@xylar Did you get a chance to run the tests again with |
|
@mcarlson801, I was away on vacation last week but I'm looking at this now. |
|
@mcarlson801, I'm very sorry for the additional delay on this. I ran the full integration suite after rebuilding the trilinos and albany spack packages as you suggested ( All non-restart tests are now passing. Restart tests all seem to have small diffs (as reported above): As before, a typical diff looks like: |
@xylar Awesome that everything is running now, thanks! The restart failures will take some digging on our end and probably won't happen right away. Do we need those tests passing before this can be merged? If so, we can have our automated pm-gpu testing use this branch temporarily for tracking purposes until we get it fixed. |
|
@mcarlson801, what we need for this to be merged is a new spack build not just on Perlmutter but on all machines where Compass is supported (we can't update one location without updating all). That will require having E3SM-Project/spack#31 merged, and making sure those changes are also included in the corresponding branch for the latest release of the |
|
Since we need to rebuild all spack environments to bring in this feature, #857 will replace this PR. |
Checklist
api.rst) has any new or modified class, method and/or functions listedTestingin this PR) any testing that was used to verify the changes